[HMA] Enabling Skipping SWA for Fp8 Quant #24912
Conversation
Signed-off-by: Jonas Kuebler <[email protected]>
|
This pull request has merge conflicts that must be resolved before it can be |
There was a problem hiding this comment.
Does the different block_size of swa and full attn compatible with prefix caching? We only have the BlockHash for one block_size. My plan is to combine the kv cache of two full attn layers, so that it has the same size with one sliding window layer. I'm prototyping.
robertgshaw2-redhat
changed the title
@heheda12345 Thanks a lot for looking into this. Great catch. I overlooked that as all the perf benchmarking I did was disabling prefix caching. But yeah, when enabling prefix caching it breaks because of non-uniform blocks. I was focusing on ensuring that the page size is uniform. If we also need to ensure that block size is uniform I also cannot think of a way other than combining blocks... |
|
Here is my PR for supporting different data type by adjusting block_size. Can you have a try? #24949 |
…_quant # Conflicts: # vllm/v1/core/kv_cache_utils.py
@heheda12345 Probably makes sense to update my PR only once yours is merged? |
Sounds good. |
|
This pull request has merge conflicts that must be resolved before it can be |
[WIP] see #24916 for context
Known limitations
Currently not compatible w/ prefix caching because we use two different block sizes for SW and Full Attention layers.Purpose
This PR adds a flag to skip the sliding window layers when quantizing the attention. The idea is that we can anyways not save much memory / latency in those layers, yet quantizing adds overheads. Furthermore, quantization always comes with additional quality risks. Not quantizing the SW layers leads to better performance as outlined in #24916
Turing off all SW layers gives us two groups of attention configs:
We currently use 2x the block size for Full attention in FP8 such that the page size equals between both groups. However, as pointed out by @heheda12345 , this currently makes it incompatible w/ prefix caching (requiring uniform block size).
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.